Image processing apparatus, image processing method, and program and recording medium used therewith

ABSTRACT

An image processing apparatus acquires and transforms first plane image data representing a space having a depth. The apparatus includes a vanishing point estimating unit for estimating a vanishing point of the first plane image data, an angle-of-view estimating unit for estimating an angle of view of the first plane image data, and an image generating unit for generating, on the basis of the vanishing point estimated by the vanishing point estimating unit and the angle of view estimated by the angle-of-view estimating unit, second plane image data corresponding to a case in which the first plane image data is projected onto a portion corresponding to the angle of view on a curved surface of a cylinder having a predetermined radius by using, as a reference point, a position being the center of the cylinder and being equal in height to the vanishing point.

CROSS REFERENCES TO RELATED APPLICATIONS

The present invention contains subject matter related to Japanese PatentApplication JP 2005-133174 filed in the Japanese Patent Office on Apr.28, 2005, the entire contents of which are incorporated herein byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to image processing apparatuses andmethods, and programs and recording media used therewith, and, inparticular, to an image processing apparatus and method for providing anatural retinal image to an observer at an unspecified position, and aprogram and recording medium used therewith.

2. Description of the Related Art

In order to use a large screen to display realistic video whose size isa life size of an observer or greater, typically, the observer needs tostand at a predetermined position in front of the screen, or thedisplayed video is transformed depending on the position of theobserver, whereby the observer, who observes the screen, can obtain acorrect (natural) retinal image that is realistic. The correct retinalimage means that the retinal image obtained such that the observerobserves the video displayed on the screen is substantially equivalentto a retinal image obtained when the observer is present in an actualscene.

Technologies for displaying realistic video for observers include, forexample, a so-called “immersive display apparatus”. An example using animmersive display apparatus is a system in which, by disposing largescreens in a space around a user and projecting video from behind thescreens by using projectors, the user can have various virtualexperiences in the space. Regarding this system, see, for example,Cruz-Neira, C., Sandin, D. J., and DeFanti, T. A., Surround-ScreenProjection-Based Virtual Reality: The Design and Implementation of theCAVE, Proceedings of SIGGRAPH '93, pp. 135-142, 1993.

SUMMARY OF THE INVENTION

In a case of the related art in which a large screen is used to providean observer with realistic video, as described above, the observer needsto stand at a predetermined position in front of the screen, or thedisplayed video needs to be transformed depending on the position of theobserver. In other words, in a technology of the related art, only videothat is observed from a certain position is designed to provide anobserver with a correct retinal image. Therefore, in the technology ofthe related art, for example, when a plurality of observerssimultaneously observe the video, it is difficult to simultaneouslyprovide a correct retinal image to each observer.

The present invention has been made in view of such circumstances. It isdesirable to provide a natural retinal image to an observer at anunspecified position in front of (within a range equal to the width of)displayed video or image.

According to an embodiment of the present invention, there is provided afirst image processing apparatus for acquiring and transforming firstplane image data representing a space having a depth. The imageprocessing apparatus includes vanishing point estimating means forestimating a vanishing point of the first plane image data,angle-of-view estimating means for estimating an angle of view of thefirst plane image data, and image generating means for generating, onthe basis of the vanishing point estimated by the vanishing pointestimating means and the angle of view estimated by the angle-of-viewestimating means, second plane image data corresponding to a case inwhich the first plane image data is projected onto a portioncorresponding to the angle of view on a curved surface of a cylinderhaving a predetermined radius by using, as a reference point, a positionbeing the center of the cylinder and being equal in height to thevanishing point.

Preferably, the angle-of-view estimating means estimates the angle ofview on the basis of the vanishing point estimated by the vanishingpoint estimating means.

The vanishing point estimating means may include drawing means fordrawing perspective lines of the first plane image data, and vanishingpoint extracting means for extracting the vanishing point of the firstplane image data on the basis of the perspective lines.

The vanishing point estimating means includes feature value extractingmeans for extracting feature values of the first plane image data,feature value quantizing means for quantizing the feature valuesextracted by the feature value extracting means,feature-value-distribution calculating means for calculating, on thebasis of the feature values quantized by the feature value quantizingmeans, a feature value distribution indicating a type of gradient withwhich the feature values are distributed in a vertical direction of thefirst plane image data, and vanishing point extracting means forextracting the vanishing point of the first plane image data on thebasis of the feature value distribution calculated by thefeature-value-distribution calculating means.

The angle-of-view estimating means may include plane view generatingmeans for generating a plane view obtained by assuming that the spacerepresented by the first plane image data is vertically viewed fromabove, and angle-of-view calculating means for calculating the angle ofview by detecting the position of a viewpoint in the space representedby the first plane image data in the plane view generated by the planeview generating means.

The plane view generating means may generate the plane view bycalculating an elevation angle of the viewpoint in the space representedby the first plane image data.

The image generating means may determine the radius of the cylinder onthe basis of the angle of view estimated by the angle-of-view estimatingmeans and an image size of the first plane image data.

The image processing apparatus may further include display means fordisplaying the second plane image data generated by the image generatingmeans, and the display means may include a planar display.

According to another embodiment of the present invention, there isprovided a first image processing method for an image processingapparatus for acquiring and transforming first plane image datarepresenting a space having a depth. The image processing methodincludes the steps of estimating a vanishing point of the first planeimage data, estimating an angle of view of the first plane image data,and, on the basis of the vanishing point estimated by the vanishingpoint estimating means and the angle of view estimated by theangle-of-view estimating means, generating second plane image datacorresponding to a case in which the first plane image data is projectedonto a portion corresponding to the angle of view on a curved surface ofa cylinder having a predetermined radius by using, as a reference point,a position being the center of the cylinder and being equal in height tothe vanishing point.

According to another embodiment of the present invention, there isprovided a first program or program recorded on a recording medium forallowing a computer to execute processing for acquiring and transformingfirst plane image data representing a space having a depth, the programcomprising the steps of estimating a vanishing point of the first planeimage data, estimating an angle of view of the first plane image data,and, on the basis of the vanishing point estimated by the vanishingpoint estimating means and the angle of view estimated by theangle-of-view estimating means, generating second plane image datacorresponding to a case in which the first plane image data is projectedonto a portion corresponding to the angle of view on a curved surface ofa cylinder having a predetermined radius by using, as a reference point,a position being the center of the cylinder and being equal in height tothe vanishing point.

In the first image processing apparatus, image processing method, andprogram, a vanishing point of first plane image data is estimated, anangle of view of the first plane image data is estimated, and secondplane image data is generated on the basis of the estimated vanishingpoint and angle of view. The generated second plane image data is suchthat the first plane image data is projected onto a portioncorresponding to the angle of view on a curved surface of a cylinderhaving a predetermined radius by using, as a reference point, a positionbeing the center of the cylinder and being equal in height to thevanishing point.

According to another embodiment of the present invention, there isprovided a second image processing apparatus for acquiring andtransforming first plane image data representing a space having a depth.The image processing apparatus includes input means for inputting avanishing point and angle of view of the first plane image data, andimage generating means for generating, on the basis of the vanishingpoint and angle of view input by the input means, second plane imagedata corresponding to a case in which the first plane image data isprojected onto a portion corresponding to the angle of view on a curvedsurface of a cylinder having a predetermined radius by using, as areference point, a position being the center of the cylinder and beingequal in height to the vanishing point.

Preferably, the image processing apparatus further includes displaymeans for displaying the second plane image data generated by the imagegenerating means, and the display means includes a planar display.

According to another embodiment of the present invention, there isprovided a second image processing method for an image processingapparatus for acquiring and transforming first plane image datarepresenting a space having a depth. The second image processing methodincludes the steps of controlling inputting of a vanishing point andangle of view of the first plane image data, and, on the basis of thevanishing point and angle of view whose inputting is controlled in theinput control step, generating second plane image data corresponding toa case in which the first plane image data is projected onto a portioncorresponding to the angle of view on a curved surface of a cylinderhaving a predetermined radius by using, as a reference point, a positionbeing the center of the cylinder and being equal in height to thevanishing point.

In the second image processing apparatus and image processing method,after a vanishing point of first plane image data and an angle of viewof the first plane image data are input, on the basis of the inputvanishing point and plane image data, second plane image data isgenerated which corresponds to a case in which the first plane imagedata is projected onto a portion corresponding to the angle of view on acurved surface of a cylinder having a predetermined radius by using, asa reference point, a position being the center of the cylinder and beingequal in height to the vanishing point.

According to the embodiments of the present invention, a first planeimage can be transformed into a second plane image. In particular, onthe basis of the first plane image, which represents a space having adepth, the second plane image can be generated. By using the secondplane image, a natural retinal image can be provided to an observer whoobserves at an unspecified position.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of a curved surface screen and a plane screen;

FIG. 2 is a block diagram illustrating an example of the configurationof an image processing apparatus to which an embodiment of the presentinvention;

FIG. 3 is a block diagram illustrating a first example of the vanishingpoint estimating unit shown in FIG. 2;

FIG. 4 is an illustration of an example of a supplied image;

FIG. 5 is an illustration of parallel lines in a depth direction and avanishing point;

FIG. 6 is a block diagram illustrating a second example of the vanishingpoint estimating unit shown in FIG. 2;

FIG. 7 is an illustration of a perpendicularly-viewed image;

FIG. 8 is an illustration of an image captured at a predeterminedelevation angle;

FIG. 9 is an illustration of quantized data;

FIG. 10 is an illustration of the average value of distances of texture;

FIG. 11 is a graph illustrating the average value of distances oftexture;

FIG. 12 is a graph illustrating a vanishing point calculated on thebasis of the average value of distances of texture;

FIG. 13 is an illustration of a vanishing point calculated on the basisof the average value of distances of texture;

FIG. 14 is a block diagram illustrating an example of the configurationof the angle-of-view estimating unit shown in FIG. 2;

FIG. 15 is an illustration of horizontal parallel lines;

FIG. 16 is an illustration of extraction of parallel lines in a depthdirection which are parallel to diagonals;

FIG. 17 consists of illustrations of generation of a tiled image and anestimated angle of view;

FIG. 18 consists of illustrations of a case in which an error occurs indrawing parallel lines in a depth direction;

FIG. 19 is a block diagram illustrating an example of the configurationof the horizontal parallel line drawing section shown in FIG. 14;

FIG. 20 is an illustration of an example of a subject;

FIG. 21 is an illustration of the principle of image transformation;

FIG. 22 is an illustration of an image captured for the subject shown inFIG. 20;

FIG. 23 is an illustration of the principle of image transformation on acaptured image and a perpendicularly-viewed image;

FIG. 24 is an illustration of the principle of image transformation on acaptured image and a perpendicularly-viewed image;

FIG. 25 is an illustration of the principle of image transformation on acaptured image and a perpendicularly-viewed image;

FIG. 26 is an illustration of the principle of image transformation on acaptured image and a perpendicularly-viewed image;

FIG. 27 has graphs of the principle of image transformation on acaptured image and a perpendicularly-viewed image;

FIG. 28 is a graph illustrating the principle of image transformation ona captured image and a perpendicularly-viewed image;

FIG. 29 is a block diagram illustrating an example of the configurationof the image transforming unit shown in FIG. 2;

FIG. 30 is an illustration of an image-forming screen for an ordinaryimage and a virtual image-forming screen;

FIG. 31 is a graph of correspondences in pixel position between twoscreens;

FIG. 32 is a graph of correspondences in pixel position between twoscreens;

FIG. 33 is an illustration of the positions of an image-forming screenfor an ordinary image and a virtual image-forming screen;

FIG. 34 is a flowchart illustrating a process of the image processingapparatus;

FIG. 35 is a flowchart illustrating a first vanishing-point-estimatingprocess;

FIG. 36 is a flowchart illustrating a second vanishing-point-estimatingprocess;

FIG. 37 is a flowchart illustrating a reference value determiningprocess;

FIG. 38 is an illustration of determination of a reference value in thecase of specifying texture;

FIG. 39 is an illustration of determination of a reference value in thecase of specifying texture;

FIG. 40 is an illustration of determination of a reference value in thecase of specifying texture;

FIG. 41 is a flowchart illustrating an angle-of-view estimating process;

FIG. 42 is a flowchart illustrating a horizontal-parallel-line drawingprocess;

FIG. 43 is a flowchart illustrating an image transforming process;

FIG. 44 is an illustration of an image changed on the basis of an angleof view correctly estimated by applying an embodiment of the presentinvention;

FIGS. 45A and 45B are illustrations of an image changed on the basis ofan angle of view correctly estimated by applying an embodiment of thepresent invention;

FIG. 46 is an illustration of an image changed on the basis of anincorrectly estimated angle of view;

FIG. 47 is an illustration of an image changed on the basis of anincorrectly estimated angle of view;

FIGS. 48A, 48B, and 48C are illustrations of angles of view and focaldistances in FIGS. 44, 46, and 47;

FIG. 49 is an illustration of a case in which image capturing in 360degrees around a user is performed;

FIGS. 50A, 50B, 50C, and 50D are illustrations of images obtained byperforming image capturing in four directions in FIG. 49;

FIG. 51 is an illustration of an image obtained by connecting the imagesshown in FIGS. 50A to 50D;

FIG. 52 is an illustration of an image generated by connecting imagesobtained by performing image capturing in eight directions in FIG. 49;

FIG. 53 is an illustration of an image generated by applying anembodiment of the present invention; and

FIG. 54 is a block diagram showing the configuration of a personalcomputer.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Before describing an embodiment of the present invention, thecorrespondence between the features of the claims and the specificelements disclosed in an embodiment of the present invention isdiscussed below. This description is intended to assure that embodimentssupporting the claimed invention are described in this specification.Thus, even if an element in the following embodiments is not describedas relating to a certain feature of the present invention, that does notnecessarily mean that the element does not relate to that feature of theclaims. Conversely, even if an element is described herein as relatingto a certain feature of the claims, that does not necessarily mean thatthe element does not relate to other features of the claims.

Furthermore, this description should not be construed as restrictingthat all the aspects of the invention disclosed in the embodiments aredescribed in the claims. That is, the description does not deny theexistence of aspects of the present invention that are described in theembodiments but not claimed in the invention of this application, i.e.,the existence of aspects of the present invention that in future may beclaimed by a divisional application, or that may be additionally claimedthrough amendments.

The image processing apparatus (e.g., the image processing apparatus 41in FIG. 2) according to an embodiment of the present invention acquiresand transforms first plane image data (e.g., the image 81 in FIG. 4)representing a space having a depth. The image processing apparatusaccording to the embodiment of the present invention includes vanishingpoint estimating means (e.g., the vanishing point estimating unit 52 inFIG. 2) for estimating a vanishing point (e.g., the vanishing point inFIG. 13 or FIG. 5) of the first plane image data, angle-of-viewestimating means (e.g., the angle-of-view estimating unit 53 in FIG. 2)for estimating an angle of view of the first plane image data, and imagegenerating means (e.g., the image transforming unit 54 in FIG. 2) forgenerating, on the basis of the vanishing point estimated by thevanishing point estimating means and the angle of view estimated by theangle-of-view estimating means, second plane image data corresponding toa case in which the first plane image data is projected onto a portioncorresponding to the angle of view on a curved surface of a cylinderhaving a predetermined radius (e.g., “a” in FIG. 31 or FIG. 32) byusing, as a reference point, a position (e.g., position (0, 0, −a) inFIG. 31 or FIG. 32) being the center of the cylinder and being equal inheight to the vanishing point.

The vanishing point estimating means (e.g., the vanishing pointestimating unit 52-1 in FIG. 3) may include drawing means (e.g., thedepth-direction parallel-line extracting unit 71 in FIG. 3) for drawingperspective lines of the first plane image data, and vanishing pointextracting means (e.g., the vanishing point calculating unit 72 in FIG.3) for extracting the vanishing point of the first plane image data onthe basis of the perspective lines.

The vanishing point estimating means (e.g., the vanishing pointestimating unit 52-2 in FIG. 6) may include feature value extractingmeans (e.g., the feature value extracting section 131 in FIG. 6) forextracting feature values of the first plane image data, feature valuequantizing means (e.g., the quantizing section 132 in FIG. 6) forquantizing the feature values extracted by the feature value extractingmeans, feature-value-distribution calculating means (e.g., the texturegradient calculating section 133 in FIG. 6) for calculating, on thebasis of the feature values quantized by the feature value quantizingmeans, a feature value distribution (e.g., a texture gradient)indicating a type of gradient with which the feature values aredistributed in a vertical direction of the first plane image data, andvanishing point extracting means (e.g., the vanishing point calculatingsection 134 in FIG. 6) for extracting the vanishing point of the firstplane image data on the basis of the feature value distributioncalculated by the feature-value-distribution calculating means.

The angle-of-view estimating means may include plane view generatingmeans (e.g., the tiled image generating section 173 in FIG. 14) forgenerating a plane view (e.g., the tiled image in FIG. 17) obtained byassuming that the space represented by the first plane image data isvertically viewed from above, and angle-of-view calculating means (e.g.,the angle-of-view calculating section 174 in FIG. 14) for calculatingthe angle of view by detecting the position (the camera position in FIG.17) of a viewpoint in the space represented by the first plane imagedata in the plane view generated by the plane view generating means.

The plane view generating means may generate the plane view by (e.g.,processing by the horizontal parallel line drawing section 171 describedwith reference to FIG. 19) calculating an elevation angle of theviewpoint in the space represented by the first plane image data.

The image processing apparatus according to the embodiment of thepresent invention may further include display means for displaying thesecond plane image data generated by the image generating means, and thedisplay means may include a planar display (e.g., the image-displayingplanar display in FIG. 44 or FIG. 45, or a planar display for displayingthe image 501 in FIG. 53).

The image processing method according to another embodiment of thepresent invention is used for an image processing apparatus (e.g., theimage processing apparatus 41 in FIG. 2) that acquires and transformsfirst plane image data (e.g., the image 81 in FIG. 4) representing aspace having a depth. This image processing method includes the steps ofestimating (e.g., step S2 in FIG. 34) a vanishing point (e.g., thevanishing point in FIG. 13 or FIG. 5) of the first plane image data,estimating (e.g., step S3 in FIG. 34) an angle of view of the firstplane image data, and generating (e.g., step S4 in FIG. 34), on thebasis of the vanishing point estimated by the vanishing point estimatingmeans and the angle of view estimated by the angle-of-view estimatingmeans, second plane image data corresponding to a case in which thefirst plane image data is projected onto a portion corresponding to theangle of view on a curved surface of a cylinder having a predeterminedradius (e.g., “a” in FIG. 31 or FIG. 32) by using, as a reference point,a position (e.g., position (0, 0, −a) in FIG. 31 or FIG. 32) being thecenter of the cylinder and being equal in height to the vanishing point.

In addition, in the program according to an embodiment of the presentinvention and the program, recorded on the recording medium, accordingto an embodiment of the present invention, (examples of) features towhich steps correspond are similar to those in the image processingmethod according to the above embodiment.

The image processing apparatus (e.g., the image processing apparatus 41in FIG. 2) according to another embodiment of the present inventionacquires and transforms first plane image data (e.g., the image 81 inFIG. 4) representing a space having a depth. This image processingapparatus includes input means for inputting a vanishing point and angleof view of the first plane image data, and image generating means (e.g.,the image transforming unit 54 in FIG. 2) for generating, on the basisof the vanishing point and angle of view input by the input means,second plane image data corresponding to a case in which the first planeimage data is projected onto a portion corresponding to the angle ofview on a curved surface of a cylinder having a predetermined radius(e.g., “a” in FIG. 31 or FIG. 32) by using, as a reference point, aposition (e.g., point (0, 0, −a) in FIG. 31 or FIG. 32) being the centerof the cylinder and being equal in height to the vanishing point.

The above image processing apparatus may further include display meansfor displaying the second plane image data generated by the imagegenerating means, and the display means may include a planar display(e.g., the image-displaying planar display in FIG. 44 or FIG. 45, or aplanar display for displaying the image 501 in FIG. 53).

The image processing apparatus according to another embodiment of thepresent invention is used for an image processing apparatus (e.g., theimage processing apparatus 41 in FIG. 2) that acquires and transformsfirst plane image data (e.g., the image 81 in FIG. 4) representing aspace having a depth. This image processing method includes the steps ofcontrolling inputting of a vanishing point and angle of view of thefirst plane image data, and, on the basis of the vanishing point andangle of view whose inputting is controlled in the input control step,generating (e.g., step S4 in FIG. 34) second plane image datacorresponding to a case in which the first plane image data is projectedonto a portion corresponding to the angle of view on a curved surface ofa cylinder having a predetermined radius (e.g., “a” in FIG. 31 or FIG.32) by using, as a reference point, a position (e.g., position (0, 0,−a) in FIG. 31 or FIG. 32) being the center of the cylinder and beingequal in height to the vanishing point.

Embodiments of the present invention are described below with referenceto the accompanying drawings.

An image processing apparatus to which an embodiment of the presentinvention is applied generates a plane image or video within apredetermined angle of view which is captured by an ordinary camera. Inother words, the image processing apparatus generates a plane image orvideo in which, when an observer observes the plane image or video froma plurality of viewpoints, the observer does not notice anyunnaturalness.

For example, as shown in FIG. 1, onto a cylindrical surface includingcurved surface portion 1, an image or video can be projected so that,when an observer 11 who exists at the center of the cylinder whilehorizontally rotating 360 degrees, the observer does not notice anyunnaturalness at any angle.

After transforming a plane image or moving picture (video) captured byan ordinary camera, the moving picture including a plurality of planeimages, into an image or video obtained when the plane image or movingpicture is projected onto the curved surface portion 1, the imageprocessing apparatus to which the embodiment of the present invention isapplied can display the obtained image or video on a plane 2 or canprint out the obtained image or video on a similar plane. On the basisof the image or video displayed on the plane 2, when a plurality ofobservers 12 to 14 in front of the plane 2 observe the front from theirpositions (when lines of sight of the observers 12 to 14 areperpendicular to the plane 2 or each have an angle close toperpendicularity), they can obtain correct retinal images similar tothose obtained when the observer 11 at the center of the cylinderincluding the curved surface portion 1 changes its direction (angle) tothe plane 2.

Image processing is exemplified below. However, obviously, byimplementing, on all frame images, processing to which an embodiment ofthe present invention is applied, an embodiment of the present inventionis applicable to even a case in which video (moving picture) isdisplayed assuming that the video includes a plurality of frame images(still images).

FIG. 2 is a block diagram showing an example of the configuration of animage processing apparatus 41 to which an embodiment of the presentinvention is applied.

The image processing apparatus 41 includes an image data acquiring unit51, a vanishing point estimating unit 52, an angle-of-view estimatingunit 53, an image transforming unit 54, and an image data output unit55.

The image data acquiring unit 51 acquires a plane image captured by anordinary camera, that is, image data corresponding to an ordinary planeimage captured by using a lens position of the camera as a viewpoint onthe basis of the principle of a so-called “pinhole camera”, and suppliesthe acquired image data to the vanishing point estimating unit 52.

The vanishing point estimating unit 52 detects a vanishing point of thesupplied image data, and supplies information of the detected vanishingpoint and the supplied image data to the angle-of-view estimating unit53. The vanishing point is a point at which, when parallel lines in athree-dimensional space are projected onto an image plane by perspectivetransformation, straight lines on the image plane which correspond tothe parallel lines converge. In other words, the vanishing point is an“infinitely far point” on a plane image onto which a space actuallyhaving a depth is projected. The vanishing point is recognized as apoint in which intersections of extended lines of parallel lines (e.g.,in an image of a room, its ridge lines) in a depth direction andextensions of planes (e.g., in an image of a room, planes correspondingto a floor, walls, and a ceiling extending in the depth direction)extending in the depth direction converge in an infinite direction. Astraight line horizontally drawn from the vanishing point in the imageis the horizon. Thus, estimation of the vanishing point is, in otherwords, estimation of the horizon. Estimation of the vanishing point bythe vanishing point estimating unit 52 may be performed by any method.Two types of vanishing point estimating methods are described below asspecific examples with reference to FIGS. 3 to 13.

The angle-of-view estimating unit 53 estimates an angle of view of thesupplied image data, that is, an angle representing an image-capturingrange of the camera used for capturing the image data, and supplies theestimated angle of view and the supplied image data to the imagetransforming unit 54. Estimation of the angle of view by theangle-of-view estimating unit 53 may be performed by any method. Aspecific example of the method is described later with reference toFIGS. 14 to 28.

The image transforming unit 54 transforms the supplied image data on thebasis of the estimated angle of view supplied from the angle-of-viewestimating unit 53, and supplies the transformed image data to the imagedata output unit 55. Details of the process of the image transformingunit 54 are described below with reference to FIGS. 29 to 33.

The image data output unit 55 executes processing such as outputting anddisplaying the transformed image data on a planar large display,printing out the transformed image data, recording the transformed imagedata, and transmitting the transformed image data to another apparatusthrough a predetermined communication medium.

When the vanishing point of the supplied image data is known beforehand,by using information of the vanishing point known beforehand for theestimation of the angle of view, the estimation of the angle of view bythe vanishing point estimating unit 52 can be omitted. In addition, whenthe angle of view of the supplied image data is known beforehand, byusing information of the angle of view known beforehand for thetransformation of the image data by the image transforming unit 54, theestimation of the angle of view by the angle-of-view estimating unit 53can be omitted.

In this case, by providing the image processing apparatus 41 shown inFIG. 2 with a data input unit (not shown), the data input unit canreceive input data of the vanishing point or angle of view and cansupply the input data to the image transforming unit 54.

FIG. 3 is a block diagram showing the configuration of a vanishing pointestimating unit 52-1 which uses a first vanishing-point estimatingmethod to estimate a vanishing point and which is a first example of thevanishing point estimating unit 52 shown in FIG. 1.

By using, for example, an edge filter to extract straight linecomponents from the supplied image data, a depth-direction parallel-lineextracting unit 71 extracts so-called “perspective lines” which areparallel to the ground (horizontal plane) and which extend from thefront to back side of the image (in a direction identical to thedirection of the viewpoint of the camera). When image data correspondingto the image shown in FIG. 4 is supplied, straight lines that are to beextracted include, for example, ridge lines of the room, and the grainof flooring of the floor. For only finding a vanishing point, it is onlynecessary to draw a plurality of perspective lines. However, on thebasis of the extracted straight lines (perspective lines), thedepth-direction parallel-line extracting unit 71 draws a plurality ofparallel lines that are disposed in an actual space at equal intervalson a predetermined plane such as the ground or a floor so that they aresuitable for processing (described later). In other words, thedepth-direction parallel-line extracting unit 71 draws parallel lines(perspective lines) at equal intervals on a predetermined plane in aspace indicated by a plane image. The drawn parallel lines arehereinafter referred to as the “parallel lines in the depth direction”.

The vanishing point calculating unit 72 finds an intersection of theparallel lines in the depth direction extracted by the depth-directionparallel-line extracting unit 71, and uses the intersection as avanishing point of this image.

In other words, when the image data acquiring unit 51 receives the inputimage 81 shown in FIG. 4, and supplies its image data to the vanishingpoint estimating unit 52-1, the depth-direction parallel-line extractingunit 71 draws a plurality of parallel lines in the depth direction, asshown in FIG. 5, and an intersection of the parallel lines in the depthdirection is used as a vanishing point by the vanishing pointcalculating unit 72. For example, when a plurality of parallel lines inthe depth direction are extracted due to a cause such as an error inextraction of parallel lines in the depth direction, the vanishing pointcalculating unit 72 may use a centroid of the intersections as avanishing point, and may use, as a vanishing point, a point at whichmost parallel lines in the depth direction intersect.

FIG. 6 is a block diagram showing the configuration of a vanishing pointestimating unit 52-2 which estimates a vanishing point by using a secondvanishing-point estimating method and which is an example of thevanishing point estimating unit 52 shown in FIG. 1.

The vanishing point estimating unit 52-2, which is described withreference to FIG. 6, estimates a vanishing point from image data byusing a so-called “texture gradient”.

By using a camera to capture, at a predetermined elevation angle, animage of a place where, in a top view, a predetermined pattern (such ascolor and luminance) extends in a substantially uniform manner, forexample, the flower garden shown in FIG. 7, the picture shown in FIG. 8in which the predetermined pattern becomes coarse at a position closerto the front side and becomes fine at a position farther from the frontside can be obtained. A unit of a pattern having a tendency in which,when an image of the pattern is captured at a predetermined elevationangle, the pattern looks coarse at a position closer to the front sideof the image and looks fine at a position farther from the front side iscalled “texture”. The degree of the tendency is called a “texturegradient”. In other words, a texture gradient is determined by anelevation angle in image capturing.

The feature value extracting section 131 extracts a feature value (e.g.,a color-difference value or edge intensity of the pixel) of each pixelof the input image. When using an edge intensity as a feature value, thefeature value extracting section 131 extracts the edge intensity byusing a built-in differential filter (not shown) to enhance an inputimage edge.

A quantizing section 132 quantizes the input image on the basis of thefeature value of each pixel extracted by the feature value extractingsection 131. For example, when the feature value is a color-differencevalue, the value of each pixel having a color-difference value equal toa predetermined reference value is set to one, and the value of eachpixel having a different color-difference value is set to zero. When theimage data of the image in FIG. 8 is quantized, quantized data as shownin, for example, FIG. 9, can be obtained.

A texture gradient calculating section 133 calculates the average valueof distances between white points (texture) in units of lines from thequantized data (e.g., the quantized data described with reference toFIG. 9) obtained by the quantizing section 132. As shown in, forexample, FIG. 10, when a plurality of pixels whose values are ones arearranged in a predetermined line, the average value of distances a₁, a₂,a₃, and a₄ therebetween is calculated.

By plotting the average values calculated in units of lines, which arerepresented by AV, as shown in FIG. 11, on the basis of positions on aY-axis in the image data, the texture gradient calculating section 133sets a regression line, as shown in FIG. 12. In other words, thisregression line corresponds to the texture gradient. The texturegradient calculating section 133 supplies the set regression line to avanishing point calculating section 134.

The vanishing point calculating section 134 calculates, as a vanishingpoint, an intersection between the set regression line and the Y-axis.

Because the vanishing point is a point at which a plane of a subjectwhen it is viewed converges in the infinite far direction, the vanishingpoint may exist at a position beyond an image area of the input image,as indicated by the vanishing point R shown in FIG. 13. The X-coordinateof the vanishing point R is the X-coordinate of the central point of theinput image.

Next, FIG. 14 is a block diagram showing an example of the configurationof the angle-of-view estimating unit 53 shown in FIG. 3.

The angle-of-view estimating unit 53 includes a horizontal parallel linedrawing section 171, a parallel line extracting section 172, a tiledimage generating section 173, and an angle-of-view calculating section174.

The horizontal parallel line drawing section 171 draws a base line whichis perpendicular in an actual subject space to the supplied parallellines in the depth direction and which indicates the width of thesubject whose image is captured, and draws parallel lines in thehorizontal direction on the basis of the base line. These parallel linesare called the “horizontal parallel lines”. The horizontal parallellines are drawn so as to be at intervals equal to those of the parallellines in the depth direction. Specifically, the horizontal parallellines are drawn so that, as shown in FIG. 15, after a base line 191 isdrawn, the horizontal parallel lines are parallel to the base line 191and distances among the horizontal parallel lines are equal to those ofthe parallel lines in the depth direction, that is, so that squares aredrawn in the subject space by the parallel lines in the depth directionand the horizontal parallel lines. At this time, by drawing only thebase line 191 and the first inner parallel line therefrom, andadditionally drawing diagonals of the formed squares, other horizontalparallel lines can be easily drawn.

When the vanishing point estimating unit 52-2 described with referenceto FIG. 6 is used as the vanishing point estimating unit 52, theparallel lines in the depth direction are not drawn. Thus, similarly tothe depth-direction parallel-line extracting unit 71, described withreference to FIG. 3, of the vanishing point estimating unit 52-1, thehorizontal parallel line drawing section 171 draws the parallel lines inthe depth direction, and subsequently draws the horizontal parallellines.

From the parallel lines in the depth direction and the diagonals of thesquares, the parallel line extracting section 172 extracts those thatare parallel to each other in the image. When the diagonals of thesquares are drawn from the bottom left to top right in the image, anyone of the diagonals on the right side of the image and any one of theparallel lines in the depth direction are parallel to each other in theimage. As shown in FIG. 16, a diagonal 201 and a depth-directionparallel line 202 are parallel to each other. Thus, the diagonal 201 andthe depth-direction parallel line 202 are extracted by the parallel lineextracting section 172. In addition, when the diagonals of the squaresare drawn from the bottom right to top left of the image, any one of thediagonals on the left side of the image and any one of the parallellines in the depth direction on the right side of the image are parallelto each other in the image.

The tiled image generating section 173 generates a tiled image on thebasis of the parallel lines in the depth direction and the horizontalparallel lines. Specifically, the tiled image generating section 173generates a plan view obtained when an area including the subject andthe camera is viewed from the top, and draws, in the plan view, as shownin FIG. 17, the parallel lines in the depth direction and the horizontalparallel lines in tiled (grided) form. The plan view in which theparallel lines in the depth direction and the horizontal parallel linesare drawn in tiled form is hereinafter referred to as the “tailedimage”.

The angle-of-view calculating section 174 calculates the angle of viewof the camera on the basis of the tiled image generated by the tiledimage generating section 173, the base line 191 drawn by the horizontalparallel line drawing section 171, and a pair of the depth-directionparallel line and the diagonal of the square which are extracted by theparallel line extracting section 172. In other words, as shown in FIG.17, the angle-of-view calculating section 174 draws, in the tiled image,the diagonal 201 and depth-direction parallel line 202 extracted by theparallel line extracting section 172. The distance between theintersection of the diagonal 201 and the depth-direction parallel line202 in the tiled image and the base line 191 corresponds to a depthdistance between the camera used for capturing this image and the focusof the camera. In other words, it can be assumed that, in the tiledimage, the intersection of the diagonal 201 and the depth-directionparallel line 202 is the position of a camera 221. The assumed positionof the camera 221 corresponds to a viewpoint (viewpoint of the observerwhen the image 81 is a correct retinal image) in an image 81. In thetiled image, an angle between straight lines drawn from the camera 221to ends of the base line 191 is a calculated angle of view.

In the case of FIG. 17, the angel of view, represented by fov, iscalculated as approximately 57 degrees.

Although the horizontal parallel line drawing section 171 draws thehorizontal parallel lines having intervals equal to those of theparallel lines in the depth direction, when an error occurs in eachinterval of the parallel lines in the depth direction and the horizontalparallel lines, an error occurs in the angel of view calculated by theangle-of-view calculating section 174.

As shown in, for example, FIG. 18, when the horizontal parallel linesare drawn at intervals different from those in the case described withreference to FIG. 17, an angle of view calculated by using the tiledimage differs from that obtained in the case of FIG. 17.

In this case, after calculating the above-described elevation angle onthe basis of the vanishing point estimated by the vanishing pointestimating unit 52, and generating a perpendicularly-viewed image (forexample, the image shown in FIG. 7 when the input image is the imageshown in FIG. 8) corresponding to an image obtained when the input imageis one captured from the top, the horizontal parallel lines may be drawnon the basis of the perpendicularly-viewed image.

FIG. 19 is a block diagram showing an example of the configuration ofthe horizontal parallel line drawing section 171 when the elevationangle can be calculated.

The horizontal parallel line drawing section 171 can include anelevation angle calculator 251, a perpendicularly-viewed-image generator252, and a horizontal line drawer 253.

The elevation angle calculator 251 uses the coordinates of the vanishingpoint supplied from the vanishing point estimating unit 52 to calculatean elevation angle of the camera used when the input image is captured,and supplies the calculated elevation angle to theperpendicularly-viewed-image generator 252.

By using the calculated elevation angle supplied from the elevationangle calculator 251, the perpendicularly-viewed-image generator 252transforms the input image into the perpendicularly-viewed image.

Next, processing that is executed by the elevation angle calculator 251and the perpendicularly-viewed-image generator 252 is described belowwith reference to FIGS. 20 to 28.

By performing image capturing on, for example, an iron sheet T withholes therein at equal intervals, as shown in FIG. 21, at predeterminedelevation angle φ by using a camera CA, an image Da can be obtainedwhich has such a gradient that the hole pattern (texture) looks coarseat a position closer to the front side and looks fine at a positionfarther from the front side. The scale M in FIG. 22 indicates positionsin the vertical direction (texture gradient direction) of the holepattern.

Next, as shown in FIG. 23, by using a projector PJ to project the imageDa having the above texture gradient on a screen SC inclined at an angleequal to elevation angle φ of the camera CA, an image Db of the ironsheet T in which, as shown in FIG. 24, the hole pattern has regularintervals similarly to the actual iron sheet T (FIG. 20) is displayed.

The bases of the arrows shown in FIG. 24 indicate predetermined verticalpositions of the holes in the pattern of the image Da, and the tips ofthe arrows indicate vertical positions of holes in the pattern of theimage Db which correspond to the holes in the pattern of the originalimage Da.

In other words, the perpendicularly-viewed-image generator 252 generatesthe perpendicularly-viewed image by using geometric relationshipsbetween pixel positions and the elevation angle (φ) of theperpendicularly-viewed image (image Db) and input image (image Da) shownin FIG. 24 to detect pixels of the input image which correspond to thepixels of the perpendicularly-viewed image, and setting the pixel valuesof the detected pixels of the input image in the pixels of theperpendicularly-viewed image.

In the example shown in FIG. 22, the vanishing point R (accurately, avanishing point image) exists in an upper portion of the image Da inwhich the vertical intervals in the hole pattern decrease. As shown inFIG. 24, the line connecting the vanishing point R and the focus Q isparallel to a subject plane (the screen surface in the example of FIG.24). Thus, the elevation angle (angle between a camera optical axis andthe subject plane) can be obtained by calculating the vanishing point R.In other words, for this reason, the elevation angle calculator 251 cancalculate elevation angle φ by using the vanishing point R.

As shown in FIG. 25, elevation angle φ is equal to an angle formedbetween the line connecting the vanishing point R and the focus Q andthe line connecting the central point of an image forming plane Z.Elevation angle φ is calculated byφ=tan⁻¹(p/kh)=tan⁻¹(r/k)  (1)where p represents a distance between the central position and thevanishing point R in the input image shown in FIG. 13, h represents theY-axial dimension (height) of the input image shown in FIG. 13, rrepresents p/h, and K represents a predetermined coefficient.

In addition, the geometric relationships shown in FIG. 25 can be alsoshown in FIG. 26.

After the elevation angle φ is calculated as described above, theperpendicularly-viewed-image generator 252 uses the calculated elevationangle to transform the input image into the perpendicularly-viewedimage.

Specifically, as shown in FIG. 27, for each pixel, for example, pixel(x₁, y₁) of the perpendicularly-viewed image, theperpendicularly-viewed-image generator 252 detects a corresponding pixel(x₀, y₀).

The pixel (x₁, y₁) of the perpendicularly-viewed image, the pixel (x₀,y₀) of the input image, and elevation angle φ have the geometricrelationships shown in FIG. 28. Thus, the pixel (x₀, y₀) of the inputimage corresponding to the pixel (x₁, y₁) of the perpendicularly-viewedimage can be found by the following expressions:

$\begin{matrix}{{y_{0} = {\frac{y_{1}\sin\;\phi}{d + {y_{1}\cos\;\phi}}{kh}}}{x_{0} = {\frac{x_{1}}{d + {y_{1}\cos\;\phi}}{kh}}}} & (2)\end{matrix}$

By using these expressions, when the input image is supplied, an actualpattern on the subject plane can be estimated.

By drawing depth-direction horizontal parallel lines and horizontalparallel lines on the perpendicularly-viewed image generated by theperpendicularly-viewed-image generator 252, and subsequently executinginverse transformation to the transformation executed by theperpendicularly-viewed-image generator 252, the horizontal line drawer253 generates image data of an image in which, as described withreference to FIGS. 16 and 17, correct parallel lines in the depthdirection and horizontal parallel lines are drawn, and supplies theimage data to the parallel line extracting section 172.

Next, FIG. 29 is a block diagram showing an example of the configurationof the image transforming unit 54 shown in FIG. 2.

Although an ordinary image based on a pinhole camera model is designedso that the image can be formed on a plane screen, the imagetransforming unit 54 can perform transformation into an image obtainedwhen the image formed on the plane screen is formed on a screen formedby a curved surface portion included in a cylinder. In other words, theimage transforming unit 54 can perform transformation into an imageobtained when an image on a plane screen is projected onto a screenformed by a curved surface portion which is included in a cylinder andwhich corresponds to an angle of view, while using, as a referenceposition, a position which is the center of the cylinder and whichserves as a horizontal line height. For example, as shown in FIG. 30, animage captured by a camera 301 and supplied to the camera 301 isprojected onto an ordinary imaging screen 302. Unlike that, a virtualimaging screen 303 whose shape is a cutout of a cylinder is assumed. Theimage transforming unit 54 transforms the image supplied to the imageprocessing apparatus 41, that is, the image projected onto the virtualimaging screen 303, into an image that is formed on the virtual imagingscreen 303.

The source image data acquiring section 281 acquires the image suppliedfrom the angle-of-view estimating unit 53 and supplied to the imageprocessing apparatus 41, that is, image data of the image prior totransformation. The source image data acquiring section 281 supplies theacquired image data to the corresponding pixel extracting section 284,and supplies a transformed image surface generating section 282 withdimension information prior to transformation.

On the basis of the dimension information of the image prior totransformation which is supplied from the source image data acquiringsection 281, the transformed image surface generating section 282prepares a surface of an output image equal in dimension to the sourceimage and supplies the surface of the output image to thepixel-of-interest extracting section 283 and the transformed imagegenerating section 285.

The pixel-of-interest extracting section 283 extracts a pixel ofinterest from the surface of a transformed image which is supplied fromthe pixel-of-interest extracting section 283, and supplies informationof the pixel of interest to the corresponding pixel extracting section284 and the transformed image generating section 285.

For each pixel of the transformed image, which is represented by thecoordinate information supplied from the pixel-of-interest extractingsection 283, the corresponding pixel extracting section 284 calculates acorresponding pixel in the transformed image on the basis of theestimated angle of view supplied from the angle-of-view estimating unit53, and supplies the pixel value of the corresponding pixel to thetransformed image generating section 285.

Specifically, as shown in FIG. 31, when a cylindrical screen havingradius a abuts on a plane screen, the corresponding pixel extractingsection 284 finds to which of pixel positions on the plane screen eachpixel of an image formed in at least part of the cylindrical screenhaving radius a corresponds. Radius a of the cylindrical screen isdetermined by an angle of view and horizontal width of the image priorto transformation. In other words, the value of a is a value at whichtan θ=1/a when ½ of the horizontal width of the image prior totransformation is 1 and ½ of the angle of view is 0.

It is assumed that origin (0, 0, 0) in an xyz coordinate system be apoint on a crossing line between the cylindrical screen having radius aand the plane screen and be coordinates having a height (i.e., theheight of the horizontal line) equal to that of the vanishing point, andthe central point of the cylindrical screen having radius a isrepresented by (0, 0, −a). When a set of coordinates on the cylindricalscreen having radius a is represented by (θ, β) on the basis of angle θfrom a z-axis on an x-y plane and y-axial coordinate β, and the set ofcoordinates is transformed into the xyz coordinate system, the obtainedcoordinates are (a sin θ, β, A(cos θ−1)). Therefore, a set ofcoordinates corresponding to pixel (θ, β) of an image formed on thecylindrical having radius a is a point at which a straight lineconnecting central position (0, 0, −a) and coordinates (a sin θ, β,A(cos θ−1)) cross an x-y plane, and has values (a tan θ, β/cos θ, 0).

In addition, as shown in FIG. 32, when the cylindrical screen havingradius a and the plane screen abut on each other, the correspondingpixel extracting section 284 may find to which of pixel positions in atleast part of the cylindrical screen having radius “a” each pixel on theplane screen corresponds.

Similarly to the case described with reference to FIG. 31, it is assumedthat origin (0, 0, 0) in the xyz coordinate system be a point on acrossing line between the cylindrical screen having radius a and theplane screen and be coordinates being equal in height (i.e., the heightof the horizontal line) to the vanishing point. When the central pointof the cylindrical screen having radius a is represented by coordinates(0, 0, −a), coordinates on the cylindrical screen which correspond tocoordinates (X, Y, 0) on the plane screen represent a point at which astraight line connecting central point (0, 0, −a) and coordinates (X, Y,0) crosses the cylindrical screen and are represented by(sin⁻¹(X/√{square root over (a² +X ²)}),aY/√{square root over (a² +X²)},a(cos θ−1))  (3)

The transformed image generating section 285 generates and supplies thetransformed image to the image data output unit 55 by using theabove-described method to repeatedly perform an operation of copying thepixel value, supplied from the corresponding pixel extracting section284, of a corresponding pixel of the source image into the position of apixel of interest of the transformed image surface.

When comparing the method described with reference to FIG. 31 and themethod described with reference to FIG. 32, in the method described withreference to FIG. 31, finally generated image data is free from lack ofpixels. It is preferable for the corresponding pixel extracting section284 to use the method described with reference to FIG. 31 to find, foreach pixel of the transformed image, a corresponding pixel of thetransformed image.

FIGS. 31 and 32 illustrate a case in which the virtual imaging screen303 that is part of the cylinder having radius a abuts on the ordinaryimaging screen 302. However, even if transformation is performedassuming a case in which the virtual imaging screen 303 and the ordinaryimaging screen 302 do not abut on each other, the transformation issimilar to that in the above-described case. In other words, onto anarea (area in which a central angle of a sector formed by a centralpoint and a screen) corresponding to an angle of view on a cylinderhaving a predetermined radius, an image on a plane screen which has thesame angle of view may be projected. The radius of the cylinder isdetermined by the angle of view and a positional relationship betweenscreens. In addition, the size of the transformed image surface is alsodetermined by the angle of view and the positional relationship betweenthe screens.

In other words, as shown in FIG. 33, if, between an image prior totransformation which is formed on the ordinary imaging screen 302 andeach of transformed images formed virtual imaging screens 303-1 to303-4, image projection is performed so as to have the same angle ofview and viewpoint (the position of the camera 301, that is, a pointcorresponding to central point (0, 0, −a)), transformation that isbasically similar to that in the above-described case can be performed,even if the ordinary imaging screen 302 exists outside or inside thecylinder including the virtual imaging screen 303.

Next, a process executed by the image processing apparatus 41 shown inFIG. 2 is described below with reference to the flowchart shown in FIG.34.

In step S1, the image data acquiring unit 51 acquires image data priorto transformation and supplies the acquired image data to the vanishingpoint estimating unit 52.

In step S2, the vanishing point estimating unit 52 executes a firstvanishing-point-estimating process, which is described later withreference to FIG. 35, or a second vanishing-point-estimating process,which is described later with reference to FIG. 36.

In step S3, the angle-of-view estimating unit 53 executes anangle-of-view estimating process, which is described later withreference to FIG. 41.

In step S4, the image transforming unit 54 executes an imagetransforming process, which is described later with reference to FIG.43.

After calculating a vanishing point in step S1, estimating an angle ofview in step S3, and generating transformed image data by performingtransformation in step S4, in step S5, the generated transformed imagedata is supplied to the image data output unit 55 and the image dataoutput unit 55 outputs the supplied image data before the processfinishes.

As described above, the image processing apparatus 41 in FIG. 2calculates a vanishing point, estimates an angle of view, transformsimage data on the basis of the angle of view, and outputs thetransformed image data. The output image data is displayed on a largeplanar display, printed out, recorded on a predetermined recordingmedium, and conveyed through a predetermined communication medium.

Next, the first vanishing-point-estimating process executed in step S2in FIG. 34 by the vanishing point estimating unit 52-1 described withreference to FIG. 3 is described below.

In step S21, by extracting, from the image acquired by the image dataacquiring unit 51, straight lines which are parallel to the ground(horizontal plane) and which extend from the front to inner side of theimage (in a direction identical to the direction of the cameraviewpoint), the depth-direction parallel-line extracting unit 71extracts parallel lines at equal intervals in the depth direction, asdescribed with reference to FIG. 5.

In step S22, the vanishing point calculating unit 72 calculates thevanishing point by calculating an intersection on the image of theparallel lines in the depth direction which are extracted by thedepth-direction parallel-line extracting unit 71, as described withreference to FIG. 5. After that, the process returns to step S2 in FIG.34 and proceeds to step S3.

In this processing, on the basis of the parallel lines in the depthdirection of the image data, the vanishing point is estimated.

Next, the second vanishing-point-estimating process executed in step S2in FIG. 34 by the vanishing point estimating unit 52-2 described withreference to FIG. 6 is described below with reference to FIG. 36.

In step S41, the feature value extracting section 131 extracts a featurevalue of each pixel in the input image. For example, a color-differencevalue or edge intensity of each pixel is extracted as the feature value.When the edge intensity is used as the feature value, the image dataacquiring unit 51 uses a differential filter (not shown) to enhance anedge of the input image and extracts the edge intensity.

In step S42, the quantizing section 132 quantizes the feature value ofthe input image on the basis of the feature value of the pixel extractedin step S41.

When the feature value is, for example, the color-difference value, thepixel value of each pixel having a color-difference value equal to apredetermined reference value is set to have 1, and the pixel value ofeach pixel having a different color-difference value is set to have 0.

For example, the input image shown in FIG. 8 is quantized to the stateshown in FIG. 9. In FIG. 9, white points indicate pixels having values1's and black points indicate pixels having values 0's.

In step S43, in processing by the texture gradient calculating section133 and the vanishing point calculating section 134, the vanishing pointis calculated. The process returns to step S2, and proceeds to step S3.

Specifically, from the quantized data (FIG. 9) obtained in step S42, thetexture gradient calculating section 133 calculates the average value(texture gradient data) of distances between white points (texture) inunits of lines. As described with reference to, FIG. 10, when pixelshaving values 1's are arranged in a predetermined line, the averagevalue of distances a₁, a₂, a₃, and a₄ therebetween is calculated.

The texture gradient calculating section 133 sets a regression line onthe basis of the average values calculated in units of lines, which arerepresented by AV (FIG. 12), as described with reference to FIG. 12. Inother words, the regression line corresponds to a texture gradient.

The quantizing section 132 calculates, as the vanishing point, anintersection between the set regression line and the Y-axis (the Y-axison the input image).

In this processing, the vanishing point is calculated on the basis ofthe texture gradient.

In step S41 in FIG. 36, the feature value extracting section 131extracts a feature value of each pixel in the input image. As thefeature value, for example, the color-difference value and edgeintensity of the pixel is extracted.

A reference value determining process that is executed in step S41 inFIG. 36 by the feature value extracting section 131 when thecolor-difference value is used as the feature value is described below.

In step S61, the feature value extracting section 131 generates ahistogram of predetermined feature values (color-difference values inthis example) from the input image.

In step S62, the feature value extracting section 131 selects ncolor-difference values having high frequencies in the histogramgenerated in step S61.

In step S63, the feature value extracting section 131 selects onecolor-difference value from the n color-difference values selected instep S62. In step S64, the feature value extracting section 131 detectsthe positions of pixels which each have a color-difference value equalto the selected color-difference value.

As indicated by the examples shown in FIGS. 38 to 40, in step S65, adifference between the maximum and minimum values of horizontalpositions (X-axial coordinates) of the pixels whose position aredetected in step S64, and a difference between the maximum and minimumvalues of vertical positions (Y-axial coordinates) of the pixels whoseposition are detected in step S64 are calculated, and the sum of bothdifferences is calculated.

In step S66, the feature value extracting section 131 determines whetheror not all the n color-difference values selected in step S62 have beenselected in step S63. If, in step S66, it is determined that there is acolor-difference value that has not been selected yet, the processreturns to step S63, and the next color-difference value is selected.Processing in step S64 and the subsequent steps is repeatedly performed.

If, in step S66, it is determined that all the color-difference valueshave been selected, in step S67, the feature value extracting section131 selects, as a reference value, a color-difference value from whichthe maximum of the sums calculated in step S65 is obtained.

In this manner, the reference value is determined. A color-differencevalue of, for example, a color which relatively exists in large numbersin the input image and which is distributed in the entirety of the inputimage is used as the reference value, and quantization of such a coloris executed.

A case in which the feature value is the color-difference value has beenexemplified. However, for another feature value such as edge intensity,a reference value is similarly determined.

Next, the angle-of-view estimating process that is executed in step S3in FIG. 34 by the angle-of-view estimating unit 53 described withreference to FIG. 14 is described below.

In step S91, when there are parallel lines in the depth direction drawnbeforehand for finding the vanishing point, for the parallel lines inthe depth direction, the horizontal parallel line drawing section 171draws horizontal parallel lines at intervals equal to those of theparallel lines in the depth direction, as described with reference toFIGS. 15 and 16. In other words, the horizontal parallel lines are drawnso that squares are drawn in perpendicularly-viewed-image form.Alternatively, when the parallel lines in the depth direction are notdrawn beforehand, after the horizontal parallel line drawing section 171draws the parallel lines in the depth direction similarly to the casedescribed with reference to FIG. 5, for the drawn parallel lines in thedepth direction, the horizontal parallel line drawing section 171 drawsthe horizontal parallel lines at intervals equal to those of theparallel lines in the depth direction, as described with reference toFIGS. 15 and 16. In other words, the horizontal parallel lines are drawnso that squares are drawn in the perpendicularly-viewed image.

In step S92, as described with reference to FIG. 16, the parallel lineextracting section 172 extracts those that are parallel from among theparallel lines in the depth direction and diagonals of the drawnsquares.

In step S93, on the basis of the parallel lines in the depth directionand the horizontal parallel lines, the tiled image generating section173 generates the tiled image, as described with reference to FIG. 17.

In step S94, on the basis of the tiled image generated by the tiledimage generating section 173, the base line drawn by the horizontalparallel line drawing section 171, and pairs of the parallel lines inthe depth direction and diagonals of the drawn squares which areextracted by the parallel line extracting section 172, the angle-of-viewcalculating section 174 calculates the angle of view, as described withreference to FIG. 17, and the process returns to step S3 beforeproceeding to step S4.

In the above processing, the angle of view of the camera used to capturethe image prior to transformation can be estimated.

In addition, when it is difficult to draw the horizontal parallel linesat intervals equal to those of the parallel lines in the depthdirection, that is, when it is difficult to draw the horizontal parallellines so that squares are drawn in perpendicularly-viewed-image form,errors occur in each interval of the parallel lines in the depthdirection and in each interval of the horizontal parallel lines, thusresulting in an error in the angle of view calculated by theangle-of-view calculating section 174.

As described in, for example, FIG. 18, when the horizontal parallellines are drawn at intervals different from those described withreference to FIG. 17, the angle of view calculated by using the tiledimage differs from that obtained in the case shown in FIG. 17. In thiscase, by changing the configuration of the horizontal parallel linedrawing section 171 to the configuration described with reference toFIG. 18, calculating the elevation angle of the camera when the imageprior to transformation is captured, and generating theperpendicularly-viewed image, the horizontal parallel lines may be drawnso that squares are drawn in the perpendicularly-viewed image to theparallel lines in the depth direction.

Next, a horizontal-parallel-line drawing process that corresponds tostep S91 in FIG. 41 when the elevation angle is calculated to generatethe perpendicularly-viewed image is described below with reference tothe flowchart shown in FIG. 42.

In step S111, the elevation angle calculator 251 calculates theelevation angle of the camera on the basis of the vanishing pointobtained in processing by the vanishing point estimating unit 52.

Specifically, the elevation angle can be calculated using expression (1)because, as shown in FIG. 25, the elevation angle is equal to an angleformed between a straight line connecting vanishing point R and focus Qand a straight line connecting the central point of image-forming planez and focus Q.

In step S112, the perpendicularly-viewed-image generator 252 uses thecalculated elevation angle to generate the perpendicularly-viewed imageby transforming the input image.

Specifically, as described with reference to FIG. 27, for each pixel(x₁, y₁) of the perpendicularly-viewed image, corresponding pixel (x₀,y₀) of the input image is detected by the perpendicularly-viewed-imagegenerator 252.

Pixel (x₁, y₁) of the perpendicularly-viewed image, pixel (x₀, y₀) ofthe input image, and elevation angle φ have the geometric relationshipsshown in FIG. 28. Thus, pixel (x₀, y₀) of the input image thatcorresponds to pixel (x₁, y₁) of the perpendicularly-viewed image can becalculated by using expression (2). By using this manner, when the inputimage is supplied, the actual pattern on the subject plane can beestimated.

In step S113, the horizontal line drawer 253 horizontally draws parallellines in the generated perpendicularly-viewed image so that squares aredrawn in perpendicularly-viewed-image form to the parallel lines in thedepth direction drawn for finding the vanishing point.

After drawing, in the perpendicularly-viewed-image generator 252, theparallel lines in the depth direction and the horizontal parallel lines,by executing inverse transformation to the transformation executed bythe perpendicularly-viewed-image generator 252, the horizontal linedrawer 253 generates image data indicating that correct parallel linesin the depth direction and correct horizontal parallel lines are drawn,and supplies the generated image data to the parallel line extractingsection 172.

Since this processing generates the perpendicularly-viewed image on thebasis of the elevation angle, the horizontal parallel lines can be drawnso that squares are accurately drawn in perpendicularly-viewed-imageform to the parallel lines in the depth direction. Therefore, the angleof view can be estimated with good accuracy.

Next, the image transforming process that is executed in step S4 in FIG.34 by the image transforming unit 54 described with reference to FIG. 29is described below with reference to the flowchart shown in FIG. 43.

In step S131, the source image data acquiring section 281 acquires imagedata prior to transformation and supplies the acquired image data to thetransformed image surface generating section 282 and the correspondingpixel extracting section 284.

In step S132, the transformed image surface generating section 282prepares a transformed image surface having dimensions equal to those ofthe image data prior to transformation, and supplies the image surfaceto the pixel-of-interest extracting section 283 and the transformedimage generating section 285.

In step S133, the pixel-of-interest extracting section 283 extracts, asa pixel of interest, an unprocessed pixel from the image surfacesupplied from the transformed image surface generating section 282, andsupplies coordinate information of the pixel of interest to thecorresponding pixel extracting section 284 and the transformed imagegenerating section 285.

In step S134, on the basis of the estimated angle of view supplied fromthe angle-of-view estimating unit 53, by using the transformationexpression described with reference to FIG. 31 or FIG. 32, thecorresponding pixel extracting section 284 extracts a pixelcorresponding to the extracted pixel of interest from the pixels of theimage data prior to transformation, and supplies the extracted pixel tothe transformed image generating section 285.

In step S135, the transformed image generating section 285 copies thepixel value of the corresponding pixel in the source image supplied fromthe corresponding pixel extracting section 284 to the position of thepixel of interest of the image surface.

In step S136, the transformed image generating section 285 determineswhether or not pixel value copying has finished for all the pixels. If,in step S136, it is determined that the pixel value copying has notfinished for all the pixels yet, the process returns to step S133, andthe subsequent steps are repeatedly performed. If, in step S136, it isdetermined that the pixel value copying has finished for all the pixels,the process returns to step S4 in FIG. 34 and proceeds to step S5.

In this processing, on the basis of an angle of view and the size ofimage data prior to transformation, each pixel of plane image data of animage captured from a predetermined camera position is replaced by anyof pixels on a virtual screen formed as at least part of a cylinder,whereby transformed image data is generated.

FIG. 44 shows an example of a transformed image generated such that theimage prior to transformation, shown in FIG. 4, is transformed on thebasis of angle of view fov₁=57 (deg) obtained as described withreference to FIG. 17. Then, a=1.086.

On the basis of the captured “interior space” image shown in FIG. 44,when an observer observes toward the front from position α shown in FIG.45A, as shown in FIG. 45B, the observer can obtain a retinal imagecorresponding to a retinal image obtained when an observer 351 in thecenter of the “interior space” looks at a direction α. On the basis ofthe captured “interior space” image shown in FIG. 44, when the observerobserves toward the front from a position β, as shown in FIG. 45B, theobserver can obtain a retinal image corresponding to a retinal imageobtained when the observer 351 in the center of the “interior space”looks at a direction β. On the basis of the captured “interior space”image shown in FIG. 44, when the observer observes toward the front froma position γ, as shown in FIG. 45B, the observer can obtain a retinalimage corresponding to a retinal image obtained when the observer 351 inthe center of the “interior space” looks at a direction γ. In otherwords, the transformed image shown in FIGS. 44 and 45A enables anobserver to obtain a correct retinal image irrespective of theobserver's positions such as the positions α to γ shown in FIG. 45A.

When the observer 351 in the center of the “interior space” observes aboundary between an inner wall and floor of the room from a lower leftcorner to lower right corner of the room while changing the angle fromdirection α to direction γ, the distance between the boundary and theobserver 351 is the largest to the lower left corner (direction α) ofthe room and to the lower right corner (direction γ) of the room, and isthe smallest to the central portion (direction β) of the room. In otherwords, the transformed image shown in FIG. 44 and FIG. 45A, the boundarybetween the inner wall and floor of the room is not a straight line buta slightly downward convex curve, and the lower left corner and lowerright corner of the room are drawn at a position higher than theboundary between the floor in the central portion and the inner wall ofthe room, that is, at a position felt as far from the observer 351.

In the transformed image, upper and lower bends that do not occur in anordinary image occur. Accordingly, it is preferable to display, for theobserver, an image in which the upper and lower bends are deleted asshown in FIG. 45A, that is, part of the image between straight lines352-1 and 352-2 in FIG. 45A.

FIG. 46 shows an example of a transformed image obtained when tanθ₂=1.0, that is, angle of view fov₂=2 tan⁻¹1.0=90 (deg) and a=2.

In addition, FIG. 47 shows an example of a transformed image obtainedwhen tan θ₂=0.25, that is, angle of view fov₃=2 tan⁻¹0.25=28.1 (deg) anda=0.5.

In other words, the transformed image shown in FIG. 44 is obtained bytransformation based on the relationship shown in FIG. 48A between angleof view fov and depth distance a between the camera and the camerafocus. The transformed image shown in FIG. 46 is obtained bytransformation based on the relationship shown in FIG. 48B between angleof view fov and depth distance a between the camera and the camerafocus. The transformed image shown in FIG. 47 is obtained bytransformation based on the relationship shown in FIG. 48C between angleof view fov and depth distance a between the camera and the camerafocus.

In other words, the transformed image shown in FIG. 46 is generated, asshown in FIG. 48B, as an image captured from a remote camera at an angleof view wider than actual for correctly obtained angle of view fov anddepth distance a (between the camera and the camera focus) in FIG. 48A.The transformed image shown in FIG. 47 is generated, as shown in FIG.48C, as an image captured from a nearby camera at an angle of viewnarrower than actual for correctly obtained angle of view fov and depthdistance a (between the camera and the camera focus) in FIG. 48A.

As described above, if image transformation is not performed on thebasis of correctly estimated angle of view and focal distance, it isdifficult to generate an image by which a correct retinal image can besupplied to an observer irrespective of positions of the observer.

Next, a remarkable advantage obtained when an embodiment of the presentinvention is applied is described below by exemplifying the case ofobtaining plane image data corresponding to a retinal image in all thedirections of 360 degrees from a predetermined point.

For example, assuming that, as shown in FIG. 49, a user 401 obtainsimage data of 360 degrees around the user 401 on a linearly extendingpavement 402 on which benches 411 and 412 are installed and around whichtrees 413-1 to 413-8 are planted, the advantage is described.

At first, FIGS. 50A to 50D show images obtained when performing imagecapturing in four directions, that is, as shown in FIG. 49, a directiona which is one side of the pavement 402 and which corresponds to thedirection of the benches 411 and 412, a direction b which differs 90degrees from the direction a and which corresponds to the direction ofthe tree 413-8, a direction c which differs 180 degrees from thedirection a, and a direction d which differs 90 degrees from thedirection c and which corresponds to the direction of the trees 413-1 to413-7.

As shown in FIG. 50A, an image 451 is a result of image capturing in thedirection a. The image 451 includes the benches 411 and 412, and, in theimage 451, the pavement 402 extends in a far side, and the boundaries ofthe pavement 402 are oblique straight lines converging at a vanishingpoint (vanishing point existing outside the image 451). As shown in FIG.50B, an image 452 is a result of image capturing in the direction b. Theimage 452 includes the tree 413-8, and the boundaries of the pavement402 look as a horizontal straight line in the image 452. As shown inFIG. 50C, an image 453 is a result of image capturing in the directionc. In the image 453, the pavement 402 extends in a far side in the image453, and the boundaries of the pavement 402 are oblique straight linesconverging at a vanishing point (vanishing point existing outside theimage 453). As shown in FIG. 50D, an image 454 is a result of imagecapturing in the direction d. The image 454 includes a plurality oftrees 413, and the boundaries of the pavement 402 look as a horizontalstraight line in the image 454.

By simply connecting the four images 451 to 454 shown in FIGS. 50A to50D, the image 471 shown in FIG. 51 can be generated. In the image 417,one tree 413, etc., which exist in a boundary portion between the imagesare corrected so as not to look as unnatural.

In the images 451 to 454 which are components of the image 471, theboundaries of the pavement 402 are shown as the straight linesconverging at the vanishing point or as the horizontal straight line.Accordingly, in the image 471, the boundaries of the pavement 402 areall formed by straight lines. Therefore, unlike a retinal image obtainedwhen the user 401 observes 360 degrees around the user 401, the image471 is formed as an unnatural image in which, at just an intermediateangle (each connecting portion of the images 451 to 454) of each of theabove angles a to d, the boundaries of the pavement 402 bend at apredetermined angle.

Conversely, for example, it is assumed that, by increasing the number ofimages captured at the position of the user 401 shown in FIG. 49, forexample, by combining and connecting a double number of images or eightimages, an image is generated.

For example, by connecting, to the connecting portions of the images 451to 454 in the image 471 described with reference to FIG. 51, imagescaptured at intermediate angles of the above angles a to d, the image481 shown in FIG. 52 can be generated. However, also in the image 481,the boundaries of the pavement 402 are formed as a set of straight linesconnected at predetermined angles in the connecting portions of theeight images.

In other words, even if, by using images obtained on the basis of theprinciple of the pinhole camera of the related art, the number ofcaptured images (image-capturing directions) is increased in order toobtain an image close to a retinal image obtained when observing 360degrees in the periphery, the boundaries of the pavement 402 are, in aprecise sense, formed as connected straight lines, so that the obtainedimage differs from a retinal image viewed by the eyes of the user 401 asan observer.

Unlike the above case, by using the image processing apparatus 41 (towhich an embodiment of the present invention is applied) to transformimages obtained on the basis of the pinhole camera principle of therelated art, and appropriately connecting the images, the boundaries ofthe pavement 402 are transformed into smooth curves, as shown in theimage 501 shown in FIG. 53. In other words, even if, from any positionon the front side of the image 501, its front is observed, a retinalimage that is supplied to the observer is substantially equivalent to aretinal image viewed by the eyes of the user 401 when the user 401rotates 360 degrees.

As described above, by applying an embodiment of the present invention,a plane image whose formation is difficult only by connecting imagesobtained on the basis of the pinhole camera principle of the relatedart, and by which a retinal image, substantially equivalent to a retinalimage viewed by the eyes of the user 401 in the actual space, can beobtained at each of viewpoints of a plurality of observers.

An embodiment of the present invention is applicable not only totransformation of an image or video captured by an ordinary camera butalso to, for example, the case of using computer graphics to createimages or video. In other words, an image or video in which a depth isrepresented by a method similar to that used for an image captured onthe basis of the pinhole camera principle may be transformed by using anembodiment of the present invention.

In addition, there is, for example, an application that enables a userto feel a virtual sense in which a person enters a screen and a screenbecomes a mirror. Specifically, there is an application that provides auser with so-called “virtual reality” or “mixed reality” although it istwo-dimensional. For example, in such virtual reality or mixed reality,video showing that fine beads and those in liquid form drop is projectedonto a large screen, and the beads and those in liquid form in thescreen hit a shadow on the screen of a user present in front of thescreen and behave as if, in the real world, the beads and those inliquid form hit the user, and, in addition, a butterfly flies into thescreen and stays in the shadow on the screen of the user.

In such applications, an embodiment of the present invention isapplicable. In other words, by applying an embodiment of the presentinvention to generation of an image or video displayed on a screen, auser that exists at an unspecified position (for example, at each of thepositions α, β, and γ in FIG. 45A) to the image or video displayed onthe screen can be supplied with a sense closer to that experienced inthe real world.

By using a transformed image generated by applying an embodiment of thepresent invention and performing predetermined image transformation onimages captured with an ordinary camera, even if an observer thatobserves the transformed image observes the front of the observer fromany position on the front side of the transformed image (in other words,if an viewpoint of the observer is perpendicular to or in asubstantially perpendicular direction to a transformed image plane), theobserver can be provided with a retinal image similar to that obtainedwhen the observer exists in the real world.

It is preferable that a display for displaying the transformed imagegenerated as described above by applying an embodiment of the presentinvention and performing predetermined image transformation on imagescaptured with an ordinary camera be larger in size.

The image transformation in the present invention is applicable not onlyto images captured with a camera or the like but also to imagesgenerated by technology such as computer graphics.

The image or video generated as described above is displayed on a largeplane screen and is printed out for observation by the observer. Inaddition to that, the image or video is applicable to various types ofapplications that use images or video.

The above-described consecutive processing may be executed either byhardware or by software. In this case, for example, the image processingapparatus 41 is formed by the personal computer 901 shown in FIG. 54.

Referring to FIG. 54, a CPU (central processing unit) 921 executesvarious types of processing in accordance with programs stored in a ROM(read-only memory) 922 or programs loaded into a RAM (random accessmemory) 923. The RAM 923 also stores data, etc, which are necessary whenthe CPU 921 executes the various types of processing, if necessary.

The CPU 921, the ROM 922, and the RAM 923 are connected to one anotherby a bus 924. The bus 924 also connects to an input/output interface925.

The input/output interface 925 connects to an input unit 926 including akeyboard and a mouse, an output unit 927 including a display formed by aCRT (cathode ray tube) or LCD (liquid crystal display), and a speaker, astorage unit 928 including a hard disk, and a communication unit 929including a modem. The communication unit 929 performs communicatingprocessing using networks including the Internet.

The input unit 926 also connects to a drive 930, if necessary, and aremovable medium 931, such as a magnetic disk, an optical disc, amagneto-optical disc, or a semiconductor memory, is loaded in the drive930, if necessary. A computer program read from the removable medium 931is installed in the storage unit 928, if necessary.

When software is used to execute the above consecutive processing,programs constituting the software are installed from a network orrecording medium to a computer built into dedicated hardware, one inwhich various functions are executed by installing various programs, forexample, a multipurpose personal computer, or the like.

Types of the recording medium include not only the removable medium 931,which is distributed separately from the personal computer 901 in orderto provide programs to a user, and which includes a program-recordedmagnetic disk (including a floppy disk), optical disc (including aCD-ROM (compact-disc read-only memory) and DVD (digital versatiledisc)), magneto-optical disc (including an MD (Mini-Disk)), orsemiconductor memory, but also the ROM 922, which is provided to theuser in a state built into the personal computer 901 and which containsprograms, and a hard disk included in the storage unit 928.

In this specification, steps constituting a program recorded on therecording medium definitely include processing steps executed in atime-series manner in given order, and include processing steps whichare executed in parallel or separately if they are not necessarilyexecuted in a time-series manner.

In addition, in this specification, the system means a logical set of aplurality of apparatuses (or functional modules for realizing particularfunctions) and is irrelevant to whether or not each apparatus and eachfunctional module are provided in a single housing.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

1. An image processing apparatus for acquiring and transforming firstplane image data representing a space having a depth, the imageprocessing apparatus comprising: means for estimating a vanishing pointbased on an imaged subject of the first plane image data, wherein thevanishing point is a point in the image data at which, when parallellines in a three-dimensional space based on objects already contained inthe image data are projected onto an image plane by perspectivetransformation, straight lines on the image plane which correspond tothe parallel lines converge; means for estimating an angle of view ofthe first plane image data; and means for generating second plane imagedata corresponding to the first plane image data projected onto aportion corresponding to the estimated angle of view of a curved surfaceof a cylinder having a predetermined radius, based on the vanishingpoint, the angle of view, and a reference point, a position of thereference point being at a center of the cylinder and at a height equalto the estimated vanishing point, wherein the means for estimating avanishing point further comprises depth-direction parallel-lineextracting means for extracting perspective lines from the imagedsubject of the first plane image data, and vanishing point calculatingmeans for calculating the vanishing point based on the perspectivelines.
 2. The image processing apparatus according to claim 1, whereinthe means for estimating a vanishing point further comprises means fordrawing perspective lines of the imaged subject of the first plane imagedata, and means for extracting the vanishing point of the first planeimage data on the basis of the perspective lines.
 3. The imageprocessing apparatus according to claim 1, wherein the means forestimating an angle of view further comprises means for generating aplane view obtained by assuming that space represented by the firstplane image data is vertically viewed from above, and means forcalculating the angle of view by detecting a position of a viewpoint inthe space represented by the first plane image data in the plane view.4. The image processing apparatus according to claim 3, wherein themeans for generating the plane view further includes means forgenerating the plane view by calculating an elevation angle of theviewpoint in the space represented by the first plane image data.
 5. Theimage processing apparatus according to claim 1, wherein the means forgenerating second plane image data further includes means fordetermining the predetermined radius of the cylinder on the basis of theangle of view and an image size of the first plane image data.
 6. Theimage processing apparatus according to claim 1, further comprisingmeans for displaying the second plane image data, wherein the means fordisplaying includes a planar display.
 7. An image processing method foran image processing apparatus for acquiring and transforming first planeimage data representing a space having a depth, the image processingmethod comprising: estimating a vanishing point based on an imagedsubject of the first plane image data, wherein the vanishing point is apoint in the image data at which, when parallel lines in athree-dimensional space based on objects already contained in the imagedata are projected onto an image plane by perspective transformation,straight lines on the image plane which correspond to the parallel linesconverge; estimating an angle of view of the first plane image data; andgenerating, by a processor, second plane image data corresponding to thefirst plane image data projected onto a portion corresponding to theestimated angle of view of a curved surface of a cylinder having apredetermined radius, based on the vanishing point, the angle of view,and a reference point, a position of the reference point being at acenter of the cylinder and at a height equal to the estimated vanishingpoint, wherein the estimating a vanishing point further includesextracting perspective lines from the imaged subject of the first planeimage data, and calculating the vanishing point based on the perspectivelines.
 8. A non-transitory computer-readable storage medium storinginstructions thereon that, when executed by an arithmetic processor,direct the arithmetic processor to execute processing for acquiring andtransforming first plane image data representing a space having a depth,the processing comprising: estimating a vanishing point based on animaged subject of the first plane image data, wherein the vanishingpoint is a point in the image data at which, when parallel lines in athree-dimensional space based on objects already contained in the imagedata are projected onto an image plane by perspective transformation,straight lines on the image plane which correspond to the parallel linesconverge; estimating an angle of view of the first plane image data; andgenerating second plane image data corresponding to the first planeimage data projected onto a portion corresponding to the estimated angleof view of a curved surface of a cylinder having a predetermined radius,based on the vanishing point, the angle of view, and a reference point,a position of the reference point being at a center of the cylinder andat a height equal to the estimated vanishing point, wherein theestimating a vanishing point further includes extracting perspectivelines from the imaged subject of the first plane image data, andcalculating the vanishing point based on the perspective lines.
 9. Animage processing apparatus for acquiring and transforming first planeimage data representing a space having a depth, the image processingapparatus comprising: a vanishing point estimating section configured toestimate a vanishing point based on an imaged subject of the first planeimage data, wherein the vanishing point is a point in the image data atwhich, when parallel lines in a three-dimensional space based on objectsalready contained in the image data are projected onto an image plane byperspective transformation, straight lines on the image plane whichcorrespond to the parallel lines converge; an angle-of-view estimatingsection configured to estimate an angle of view of the first plane imagedata; and an image generating section configured to generate secondplane image data corresponding to the first plane image data projectedonto a portion corresponding to the estimated angle of view of a curvedsurface of a cylinder having a predetermined radius, based on thevanishing point, the angle of view, and a reference point, a position ofthe reference point being at a center of the cylinder and at a heightequal to the estimated vanishing point, wherein the vanishing pointestimating section further comprises a depth-direction parallel-lineextracting unit configured to extract perspective lines from the imagedsubject of the first plane image data, and a vanishing point calculatingunit configured to calculate the vanishing point based on theperspective lines.
 10. The image processing apparatus of claim 9,wherein the vanishing point calculating unit uses a centroid ofintersections of the perspective lines as the vanishing point.
 11. Theimage processing apparatus of claim 9, wherein the vanishing pointcalculating unit uses a point at which most of the perspective linesintersect as the vanishing point.